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ABSTRACT 

The  Maritime  Operations  Center  (MOC)  was  designed  to  effectively  utilize  the 
planning  elements  of  Future  Operations  (FOPS)  to  provide  more  rapid,  accurate 
resource  allocations  consistent  with  the  vision  of  the  Commander.  MOC  staff 
simultaneously  participate  in  the  planning  effort,  while  executing  the  current 
operation,  and  supporting  headquarters  during  planning  and  execution. 
Frequently,  an  operational  planning  team  (OPT)  -  a  task-organized  team  formed 
to  conduct  integrated  planning  for  a  specific  mission  -  is  formed  by  the  MOC 
because  it  offers  the  advantage  of  a  focused  group  of  subject  matter  experts 
approaching  the  problem  in  an  integrated  manner.  However,  performance 
problems  may  be  realized  with  the  OPT  being  isolated  in  situations  that  require 
the  OPT  to  coordinate  closely  with  the  rest  of  the  MOC.  An  experiment  was 
conducted  in  which  the  MOC  planned  with  either  an  integrated  or  an  isolated 
planning  team  where  the  (1)  FOPS  team  was  supported  by  a  decision  aid / 
planning  tool  that  fosters  coordination  or  (2)  FOPS  team  used  a  planning  tool  with 
a  reduced  coordination  capability.  In  line  with  the  theme  of  this  year’s  symposium 
-  The  Evolution  of  C2  -  this  paper  describes  an  experiment  conducted  to  gain 
insight  into  advantages  and  disadvantages  associated  with  forming  an  OPT. 

INTRODUCTION 

As  part  of  the  Office  of  Naval  Research  (ONR)  Adaptive  Architectures  for  Command  and 
Control  (A2C2)  program,  the  Naval  Postgraduate  School  (NPS)  has  been  conducting  empirical 
research  to  design  and  analyze  adaptive  C2  structures  for  future  U.S.  Navy  and  Joint  forces. 
(Diedrich,  Entin,  Hutchins,  Hocevar,  Rubineau,  &  MacMillan,  2003;  Entin,  Diedrich,  Kleinman, 
Kemple,  Hocevar,  Rubineau,  &  Serfaty,  2003;  Hess,  Entin,  Hess,  Hutchins,  Kemple,  Kleinman, 
Hocevar,  &  Serfaty,  2000;  Levchuk,  Kleinman,  Pattipati,  Kemple,  &  Luoma,  2000;  and 
Hutchins,  Kemple,  Kleinman,  and  Hocevar,  2005).  Through  the  integration  of  analytical 
modeling,  human-in-the-loop  experimentation  and  computer  simulation,  this  research  has 
followed  a  “model-test-model-experiment”  paradigm  wherein  models  and  associated  simulations 
define  and  guide  experiments,  and  the  results  from  the  experiments  are  fed  back  to  improve  and 
enhance  the  models. 

The  A2C2  program  has  transitioned  its  research  results  to  the  Navy  on  a  number  of  issues.  The 
models  were  used  to  provide  input  and  support  to  several  commands  regarding  implementation 
of  FORCENet,  Expeditionary  Strike  Groups  (ESGs)  and  the  Maritime  Operations  Center 
(MOC).  For  example,  the  A2C2  models  were  used  by  the  Strategic  Studies  Group  when 
examining  ways  to  implement  FORCENet  concepts.  Two  organizational  structures  were 
designed  for  Carrier  Group  One  (CARGRU-1)  that  were  used  in  a  one-week  experiment  at  NPS 
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and  then  CARGRU-1  employed  one  of  these  structures  during  GLOBAL  99.  Following  this 
effort,  A2C2  was  asked  to  explore  the  adaptive  command  and  control  (C2)  construct  in  support 
of  the  newly  emerging  ESG  concept.  Based  upon  our  interaction  with  ADM  LeFever  the  A2C2 
project  was  asked  to  experiment  with  C2  issues  related  to  Expeditionary  Strike  Groups  (ESG). 
Based  on  validated  results  we  were  asked  to  remain  involved  in  ESG  research. 

In  recognition  of  recent  needs  for  organizational  change  within  the  Navy  the  A2C2  research 
program  has  developed  a  multi-disciplinary  research  agenda  to  conduct  experimentation  on 
issues  critical  to  Maritime  Operations  Centers  (MOC).  Objectives  for  this  year’s  empirically- 
focused  research  are  twofold,  to:  (1)  continue  our  model-based  experimentation  and  (2)  explore 
new  paradigms  for  empirical  studies  of  adaptive  C2  architectures  for  MOC  laboratory  research. 
Our  research  method  for  this  new  focus  area  on  MOC  C2  structures  and  processes  entails  (/) 
identifying  key  C2  issues  via  literature  and  attendance  at  MOC  events,  (ii)  interacting  with  the 
Fleet  and  other  subject  matter  experts,  and  (Hi)  analysis  of  selected  points  and  paths  in  the 
research  space  via  A2C2  model-driven  experimentation.  Potential  research  issues  and  questions 
include  identification  and  exploration  of  potential  “trouble  spots”  and  recommending  alternative 
organizational  structures  and  processes;  adaptation  and  scalability  across  structures  and 
processes;  and  coordination  with  external  forces  and  entities/agencies  including  information  and 
command  flows  in  combined  operations  and  coordination  between  MOCs. 

A  MOC  empirical  research  campaign  is  underway  where  the  emphasis  is  on  operational  versus 
tactical  activities,  and  planning  versus  execution.  Because  of  its  complexity,  its  mission  to 
oversee  large  operations,  and  its  dynamic  structure,  the  MOC  is  an  ideal  organization  for 
research  on  organizational  structures,  C2,  and  the  process  of  mission  planning.  Since  the  MOC 
was  designed  to  effectively  integrate  the  planning  elements  of  Current  and  Future  Operations 
(COPS  and  FOPS)  to  provide  more  rapid  and  accurate  resource  allocations  that  are  consistent 
with  mission  requirements,  our  first  experiment  focused  on  the  MOC  with  emphasis  on  intelli¬ 
gence,  surveillance,  and  reconnaissance  (ISR)  (Hutchins,  Kemple,  Kleinman,  Miller,  Pfeiffer, 
Weil,  Horn,  Puglisi,  &  Entin,  2009).  Two  alternate  structures  for  ISR  personnel  were:  (1)  central¬ 
ized  in  a  stand-alone  ISR  cell  or  (2)  decentralized  by  embedding  an  ISR  capability  into  both  the 
COPS  and  FOPS  cells. 

Teams  with  an  independent  ISR  cell  were  associated  with  more  effective  FOPS  plans  and 
exhibited  a  trend  of  higher  performance  across  most  major  performance  variables.  Teams  with  an 
independent  ISR  cell  received  the  highest  ratings  across  all  observer-based  measures  of  cell 
coordination,  cell  orientation,  cell  monitoring,  and  quality  of  actions  taken  by  cell  members. 
Teams  with  embedded  ISR  reported  significantly  higher  perceptions  of  social  cohesion  and 
mission  cohesion  than  teams  with  independent  ISR,  and  reported  higher  values  of  perceived 
teamwork  process  and  cell  efficiency.  Latencies  were  no  different  between  teams,  cells,  or  FOPS 
tasks,  and  were  not  related  to  performance  across  teams/tasks/cells.  On  this  highly-structured, 
iterative  task,  the  advantages  of  team  process  (familiarity,  cohesions,  efficacy)  that  were  gained 
by  embedding  ISR  within  FOPS  and  COPS  were  overshadowed  by  the  ability  for  COPS/ISR  to 
work  in  parallel  in  separate  battle  areas. 

Among  the  most  salient  scientific  contributions  of  the  A2C2  program  to  C2  research  is  the 
conduct  of  model-based  experimentation.  Analytical  models  are  used  to  guide  experiments  via  a 
priori  predictions  of  performance  and  process  measures  across  alternative  C2  structures,  and  by 
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conducting  sensitivity  analyses  to  suggest  values  for  design  parameters  so  that  the  experiments 
are  conducted  in  regions  where  the  dependent  variables  (DVs)  are  sensitive  to  changes  in  the 
independent  variables  (IVs).  Models  are  also  used  as  a  reference  or  guidepost  for  examining  the 
DVs  and  collected  data,  primarily  via  model-data  comparisons.  For  experiment  MOC-1 
“optimal”  resource  allocation  and  scheduling  algorithms,  developed  by  the  University  of 
Connecticut  were  used  to  determine  the  parameters  for  the  planning  task  (Mandal,  Han,  Pattipati, 
&  Kleinman,  2010).  These  included  the  number  of  assets  each  FOPS  planner  would  be 
assigning,  the  number  of  tasks  that  these  assets  would  need  to  be  assigned  to,  and  the  resource 
requirements  of  the  individual  tasks.  Model  outputs  included  the  optimal  assignment  accuracy, 
such  that  tasks  were  adjusted  so  that  the  planning  exercise  could  indeed  be  attained  with  85%+ 
accuracy.  In  turn,  this  provided  mission  guidelines  and  requirements  that  were  placed  on  the 
FOPS  planners,  as  well  as  a  way  to  evaluate  the  produced  plan  post-experiment. 

This  year  NPS  maintained  its  research  objective  to  evaluate,  via  empirical  study,  alternative  C2 
concepts  for  the  MOC  at  the  operational  level  of  war.  Our  approach  utilized  model-based 
experimentation,  with  particular  emphasis  on  simulator-embedded  models  for  decision  aids  and 
agents  to  assist  and  augment  human  participants. 

BACKGROUND 

The  MOC  is  a  distinct  functionally  organized  element  of  the  Maritime  Headquarters  which 
conducts  planning,  and  directs,  monitors,  and  assesses  functions  in  support  of  combat  across  the 
full  range  of  military  operations.  Current  Operations  (COPS)  focuses  on  monitoring  and 
assessing  the  execution  of  the  commander’s  intentions.  Future  Operations  (FOPS)  conducts 
operational-level  planning  for  near-term  operations. 

Navy  strategy  envisions  a  network  of  scalable  maritime  headquarters,  with  agility  to  transition 
between  command  roles,  and  to  deploy  forward  command  elements  rapidly  as  needed  to  meet 
the  requirements  of  the  Combatant  Commander.  Senior  Navy  leaders  have  stated  a  need  to 
refocus  and  enhance  the  Navy’s  ability  to  function  at  the  operational  level  of  war  and  to  train 
commanders  capable  of  commanding  at  that  level  (U.S.  Fleet  Forces  Command,  2007).  MOCs 
were  conceived  to  enable  these  capabilities  while  providing  a  degree  of  standardization  among 
the  maritime  headquarters  (MHQ  with  MOC  CONOPs,  2007).  The  initial  plan  establishes  MOCs 
for  each  of  the  numbered  fleets  (e.g.,  the  Seventh  Fleet  in  the  western  Pacific  Ocean,  the  sixth 
Fleet  in  the  Mediterranean,  etc.).  The  emerging  “Maritime  Headquarters  with  Maritime 
Operations  Centers”  concept  may  also  have  a  role  in  maritime  domain  awareness  (MDA),  a  key 
concept  of  the  Navy’s  new  maritime  strategy,  A  Cooperative  Strategy  for  21st  Century  Seapower 
(www.navy.mil/maritime/MaritimeStrategy.pdf). 

Navy  Operational  Planning 

Naval  forces  have  traditionally  embraced  a  fluid  form  of  fighting  known  as  maneuver  warfare 
which  causes  surprise  and  confusion  within  the  enemy  ranks  giving  naval  forces  an  advantage.  In 
major  regional  conflicts,  the  primary  warfighters  (the  geographic  combatant  commanders)  are 
the  supported  commanders.  They  are  responsible  for  carrying  out  national  tasking  as  well  as 
conducting  and  coordinating  operations  within  their  theater.  Based  on  the  magnitude  of  the  task, 
the  unified  commander  may  need  to  delegate  some  planning  to  supporting  and  subordinate 
commands.  Delegation  ensures  that  the  subordinate  staffs  who  are  most  familiar  with  the 


4 


capabilities  and  limitations  of  assigned  forces,  are  included  in  plan  development.  As  a  result, 
each  level  contributes  uniquely  to  the  plan.  Like  the  governing  operation  order,  the  naval  plan 
tests  for  adequacy,  feasibility,  and  acceptability.  Naval  staffs  are  best  able  to  determine  whether 
plans  developed  by  subordinates  can  accomplish  the  mission  and  be  executed  with  available 
resources.  We  instantiated  this  hierarchical  relationship  between  the  MOC  and  subordinate 
forces  in  the  experiment  by  having  subordinate  task  forces  as  the  lower  entity  that  MOC  planners 
sent  their  orders  to. 

Integrated  Product  Teams 

An  integrated  product  team  (IPT)  is  a  management  technique  employed  in  the  private  sector  that 
simultaneously  integrates  all  essential  acquisition  activities  through  the  use  of  multidisciplinary 
teams  to  optimize  the  design,  manufacturing,  business,  and  supportability  processes.  The  idea  is 
to  move  away  from  hierarchical  decision  making  to  a  process  where  decisions  are  facilitated 
across  organizational  structures  by  IPTs  (DoD,  1996).  The  process  is  designed  to  facilitate 
decision  making  by  making  decisions  and  recommendations  based  on  timely  input,  where  an  IPT 
replaces  the  traditional  sequential  review  and  approval  process  (DoD,  1999). 

In  1995,  then  Secretary  of  Defense  mandated  that  the  DoD  perform  as  many  acquisition 
functions  as  possible  using  IPTs.  IPTs  are  advisory  bodies  to  the  program  manager  where  a  key 
tenet  is  to  resolve  issues  and  concerns  at  the  lowest  level  possible,  and  to  expeditiously  escalate 
issues  that  need  resolution  at  a  higher  level.  Advantages  afforded  by  an  IPT  are  that  they:  (1) 
streamline  an  inefficient  process;  (2)  take  advantage  of  all  members’  expertise  and  produce  an 
acceptable  product  the  first  time;  and  (3)  focus  on  a  particular  topic.  Survey  results  indicate  the 
process  shortens  the  timeline,  reduces  life-cycle  costs,  while  continuing  to  meet  warfighter’s 
needs  (while  maintaining,  and  often  increasing,  quality)  (DoD,  1999) 

The  purpose  of  an  IPT  is  to  reach  agreement  on  a  strategy  and  plan  by  identifying  and  resolving 
issues  early,  based  upon  understanding  issues  and  the  rationale  for  the  approach  being  used.  An 
IPT  relies  on  applying  functional  expertise  in  a  team-oriented  manner  and  understanding 
customers  (command  structure,  doctrine,  tactics,  operational  environment),  and  integrating  user’s 
requirements,  logistical  requirements  and  constraints.  Principles  employed  by  IPTs  stress  a  spirit 
of  teamwork,  where  participants  are  empowered  and  authorized  to  the  maximum  extent  possible. 
These  principles  include  the  following:  (1)  Each  team  member  brings  unique  expertise  that  needs 
to  be  recognized  by  all;  (2)  Each  person’s  views  are  important  and  need  to  be  heard;  this  includes 
full  and  open  discussions,  where  cooperation  is  essential,  based  upon  reasoned  discussion;  (3)  A 
sense  of  ownership  is  key  to  success,  where  all  must  feel  their  contributions  are  important  and 
are  well  considered,  and  decisions  are  a  product  of  the  team. 

Empowerment  is  critical,  including  frequent  communication  with  their  leadership.  Qualified 
members  must  be  professionals  who  are:  (1)  current  in  their  functional  area;  (2)  knowledgeable 
in  the  mission  and  organization  they  are  representing;  (3)  trained  in  the  use  of  and  participation 
in  IPTs.  Membership  should  be  limited  to  the  minimum  essential  to  enhance  communication  and 
trust  and  participation  should  be  by  principle  member.  IPTs  should  be  considered  when  there  are 
requirements  for  multi-functional  expertise,  to  address  multi-faceted,  complex  situations  and 
issues.  The  focus  is  on  addressing  issues  that  require  balancing  cost,  schedule,  and  performance, 
and  having  clear  objectives. 
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An  underlying  premise  is  that  forming  an  IPT  with  members  who  represent  the  range  of  skills 
needed  for  the  entire  process  should  enable  teams  to  consider  more  and  a  broader  range  of 
alternatives  quickly,  and  in  a  broader  context,  and  enables  better  and  faster  decisions.  Once  on  a 
team,  the  role  of  an  IPT  member  changes  from  that  of  a  member  of  a  particular  functional 
organization,  who  focuses  on  a  given  discipline,  to  that  of  a  team  member,  who  focuses  on  a 
product  and  associated  processes. 

Operational  Planning  Teams 

Operational  Planning  Teams  (OPT)  represent  the  military’s  implementation  of  an  IPT.  When  an 
OPT  is  used  the  focus  is  on  an  integrated  approach  to  planning  (MCWP  5-1,  Marine  Corps 
Planning  Process).  Integrated  planning  does  not  replace  normal  staff  planning,  but  increases  the 
staffs  participation  by  demanding  greater  planning  input.  Principal  and  special  staffs  must  have 
representation  within  an  OPT  to  ensure  the  warfighting  functions  are  appropriately  represented 
for  integrated  planning  to  occur.  Each  primary  and  special  staff  must  be  represented  in  the  OPT, 
as  well  as  all  adjacent,  subordinate  and  supporting  commands,  and  as  required,  subject  matter 
experts  (SMEs)  are  added  to  the  team. 

Planning  by  an  OPT  considers  the  warfighting  functions  during  planning,  thus  eliminating  many 
of  the  omissions  that  proved  fatal  in  the  past,  and  is  better  able  to  visualize  the  interactions  that 
will  occur  in  execution.  For  example,  previously  the  command  element  would  recommend  a 
mission  and  default  the  planning  to  the  Ground  Control  Element  (GCE)  who  would  then  plan  in 
a  vacuum.  There  was  the  potential  for  no  consideration  of  the  capabilities  and  limitations  of  other 
elements  of  the  Marine  Air  Ground  Task  Force  (MAGTF)  during  the  GCE’s  planning.  Now  the 
Marine  Corps  fights  and  plans  as  a  MAGTF  -  the  OPT  is  at  the  heart  of  the  MAGTF ’s 
operational  planning,  supplemented  by  staff  action  and  coordination. 

Whether  for  business  (IPT)  or  military  (OPT)  the  emphasis  is  on  the  use  of  an  integrated 
approach  during  the  planning  process.  The  focus  is  on  consideration  of  capabilities  and 
limitations  of  all  other  elements  on  the  process.  Because  they  are  organized  to  be  less  hierarch¬ 
ical  during  planning  they  are  able  to  bring  all  representatives  together.  Staff  representatives  must 
keep  a  two-way  information  flow  going  between  the  OPT  and  the  staff  and  must  avoid  getting 
blind-sided  by  a  principle  staff  officer  during  a  brief  to  the  commanding  general. 

One  result  of  implementation  of  an  OPT  is  an  increase  in  situation  awareness  throughout  the 
force.  Results  of  implementing  an  OPT  are  generally  better  and  faster  decisions.  Since  MAGTFs 
fight  as  part  of  a  Joint  Task  Force  (JTF),  the  OPT  is  also  linked  to  higher,  adjacent,  and  other 
units  through  use  of  liaison  officers  (LNOs).  Therefore,  the  entire  plan  has  considered  all  the 
capabilities  and  limitations  of  the  JTF  across  the  warfighting  functions  and  throughout  the  Joint 
Area  of  Operations  (JOA).  Everyone  concerned  will  know  the  capabilities  and  limitations  of 
logistics  and  the  combat  service  support  element  (CSSE),  which  will  ensure  maximizing  the 
combat  power  of  the  force.  An  OPT  offers  benefits  that  are  analogous  to  those  provided  by  IPTs, 
that  is,  all  aspects  of  the  (design)  process  will  be  represented  so  all  these  elements  can  be 
considered  early  and  produce  a  better  system  or  plan. 

An  OPT  is  normally  formed  around  a  core  of  planners  from  either  Future  Plans  or  Future 
Operations  and  draws  members  from  both  FOPS  and  COPS.  The  OPT  is  completed  with 
representatives  from  various  other  staff  sections  and  organizations  that  can  provide  SMEs 
required  to  address  mission  requirements.  An  advantage  of  an  OPT  is  a  focused  group  of  SMEs 
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approaching  the  problem  in  an  integrated  manner,  however  the  division  of  responsibilities 
between  the  OPT  and  the  principal,  or  designated  staff  (“battle  staff’)  may  become  confused.  A 
common  misconception  is  that  an  OPT  is  viewed  as  a  substitute  for  normal  staff  action  and 
coordination.  When  this  occurs,  the  staff  tends  to  be  disengaged  from  the  planning  effort,  and  the 
OPT  is  left  unsupported  in  terms  of  staff  estimates  and  guidance.  The  result  is  that  the  OPT  may 
be  unable  to  develop  plans  that  are  complete,  supportable,  and  synchronized. 

Frequently,  an  operational  planning  team  -  a  task-organized  team  formed  to  conduct  integrated 
planning  for  a  specific  mission  -  is  formed  by  the  MOC  because  it  offers  the  advantage  of  a 
focused  group  of  subject  matter  experts  approaching  the  problem  in  an  integrated  manner. 
However,  problems  may  be  associated  with  their  being  somewhat  isolated  in  situations  that 
require  the  OPT  to  coordinate  closely  with  the  rest  of  the  MOC.  The  OPT  small  group  construct 
may  lead  to  OPT  isolation  from  the  rest  of  the  MOC  staff  despite  being  in  a  situation  requiring 
close  cross  coordination  with  other  staff  in  the  MOC.  In  line  with  the  theme  of  this  year’s 
symposium  -  The  Evolution  of  C2  -  this  paper  describes  an  experiment  conducted  to  gain 
insight  into  the  advantages  and  disadvantages  associated  with  forming  an  OPT. 

Hocevar  and  Owen  (1998)  identified  critical  issues  that  must  be  managed  to  achieve  the  desired 
outcomes  associated  with  IPTs.  They  describe  organizational-level  factors,  group-level  factors, 
and  individual  factors.  The  business  literature  on  large-scale  change  articulates  that  organizations 
are  complex  open  systems  and  as  such  change  must  simultaneously  address  structure,  technology, 
human  resources,  and  tasks  (Galbraith,  1989;  Nadler,  1981).  While  teams  provide  a  mechanism  to 
increase  flexibility  of  performance  in  the  context  of  increasing  environmental  turbulence 
(Hocevar  &  Owen,  1998),  decisions  regarding  whether  and  how  to  structure  teams  need  to 
consider  potential  limitations  that  can  result  from  the  new  structure.  Structural  changes  require 
analysis  of  tasks  and  interdependencies  as  well  as  determination  of  appropriate  integration 
mechanisms  (Hocevar  &  Owen,  1998).  Processes  must  be  analyzed  to  determine  what  sets  of 
activities  have  to  be  integrated  with  each  other  to  provide  increased  value.  In  line  with  the  goal  of 
developing  a  better  understanding  of  the  conditions  when  standing  up  an  OPT  may  produce 
undesired  side  effects,  we  are  investigating  one  factor  associated  with  formation  of  an  OPT, 
namely  how  this  structural  change  will  impact  task  performance  by  virtue  of  OPT  isolation. 

METHOD 

The  objective  of  this  experiment  was  to  examine  the  potential  problems  that  could  arise  when 
forming  an  OPT.  The  overarching  research  question  seeks  to  understand  how  are  emergent 
events  best  handled  when  resources  must  be  shared  among  separate  planning  teams?  For 
example  when  an  Operational  Planning  Team  is  formed.  The  current  study  was  designed  to 
examine  the  efficiency  and  planning  performance  of  two  alternative  organizational  structures:  (1) 
Integrated-  where  planning  teams  plan  with  a  real-time  view  of  others’  resource  planning  and 
(2)  Isolated  -  where  planning  teams  operate  in  isolation,  without  the  ability  to  directly  view 
others’  resource  planning.  The  first  experimental  hypothesis  was  that  “integrated  teams  create 
more  effective  plans  than  isolated  teams  due  to  their  real-time  awareness  that  enhances  the 
interdependent  solution.  Our  second  experimental  hypothesis  was  that  isolated  team  member 
experience  high  levels  of  workload  than  integrated  team  members  because  their  lack  of  real-time 
planning  status  requires  more  frequent  status  related  communication  in  addition  to  collaborative 
effort.  Our  third  experimental  hypothesis  was  that  isolated  team  members  communicate  more 


7 


frequently  in  response  to  emergent  events  because  isolated  team  members  must  communicate  to 
learn  how  others  alter  plans  in  response  to  unexpected  events. 

Some  MOCs  form  an  OPT  when  a  new  crisis  arises  or  a  new  unanticipated  task  needs  to  be 
accomplished.  In  these  situations,  some  MOCs  form  an  OPT,  others  do  not.  It  has  been  observed 
during  the  Naval  Staff  Officers  Course,  that  some  OPTs  tend  to  work  in  isolation,  which  can 
cause  problems  if  their  plans  overlap  -  either  in  terms  of  the  assets  needed  or  operations  -  with 
the  plans  of  the  main  body  of  the  MOC.  Our  goal  was  to  gain  insight  into  performance  differ¬ 
ences  for  teams  planning  within  a  MOC  where  they  planned  with  a  sub-team  of  the  MOC  that 
either  had  full  access  to  other  planners’  information,  i.e.,  they  were  fully  integrated,  or  the  sub¬ 
team  was  isolated  from  the  MOC.  The  isolated  case  was  meant  to  serve  as  a  surrogate  for  an 
OPT.  The  specific  task  entailed  assigning  resources  to  tasks  and  monitoring  the  effectiveness  of 
task  accomplishment  and  then  replanning  for  the  next  day  based  on  performance  at  the  end  of 
each  day. 

An  experiment  was  conducted  where  the  MOC  planned  in  the  condition  where  either  the  FOPS 
team  (1)  was  supported  by  a  decision  aid/  planning  tool  that  fosters  coordination  across  planners 
or  (2)  planned  with  a  planning  tool  with  a  reduced  coordination  capability.  These  FOPS  planners 
were  all  planning  for  use  of  the  same  resources  and  all  planning  for  mission  task  accomplishment 
at  the  same  time,  which  was  likely  to  cause  conflict  regarding  asset  availability  when  multiple 
people  request  the  same  Task  Force  to  perform  tasks.  We  hypothesized  if  the  planners  did  not 
collaborate  to  coordinate  asset  assignments  problems  would  arise. 

Participants 

Twenty- four  students  in  the  Graduate  School  of  Operational  and  Information  Sciences,  at  the 
Naval  Postgraduate  School  (NPS),  Monterey,  CA,  served  as  experimental  participants  during  a 
ten-hour  experiment  conducted  1-12  March  2010.  Their  mean  age  was  33.4  years;  services 
represented  were  Navy,  17,  Marine  Corps,  6,  and  Army,  1.  Participants’  rank  ranged  from  0-2 
to  0-6.  Six  teams  of  four  players  participated  in  the  experiment  where  each  team  consisted  of 
four  Future  Operations  (FOPS)  players  in  line  with  the  emphasis  on  planning  at  the  operational 
level  of  war  -  the  focus  for  Maritime  Operations  Centers. 

Planning  Task 

Planning  for  the  area  of  responsibility  was  divided  among  the  four  FOPS  players.  A  player  was 
responsible  for  planning  in  either  Area  A  or  Area  B  and  was  also  responsible  for  planning  for 
either  the  next  day  or  two  days  into  the  future.  This  divided  the  planning  tasks  to  make  each 
person’s  task  manageable.  It  also  resembles  the  way  planning  is  accomplished  in  a  MOC,  where 
the  FOPS  team  segments  plans  for  various  plan  phases.  Each  team  was  assigned  to  one  of  two 
experimental  conditions:  (1)  the  FOPS  team  members,  when  convened,  had  access  to  a  planning 
tool  that  displayed  planning  progress  on  all  tasks  by  all  planners  or  (2)  FOPS  team  members  had 
a  limited  version  of  the  planning  tool  that  did  not  display  planning  progress  for  other  planners. 

A  confederate  played  the  role  of  a  COPS  member  to  simulate  the  exchange  of  information  that 
occurs  between  COPS  and  FOPS.  The  COPS  player  sent  situation  reports  (SITREPS)  and 
casualty  reports  (CASREPS)  via  Chat  over  the  course  of  the  two-hour  blocks  to  provide  updates 
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on  execution  of  the  plan.  Information  in  these  SITREPS  and  CASREPS  had  to  be  considered  by 
the  FOPS  planners  as  it  impacted  the  planning  process. 

Outline  of  Events 

The  experiment  was  conducted  in  four  time  blocks,  spread  over  several  days.  Block  0  consisted 
of  an  introduction  to  the  experiment,  including  a  brief  on  the  mission,  and  initial  training  and 
practice  on  their  responsibilities.  Following  this,  FOPS  players  were  trained  in  their  role  as 
planners,  practiced  using  their  respective  planning  and  monitoring  tools,  and  communicating  via 
Chat. 

Block  1  comprised  the  first  2-hour  set  of  the  experimental  task  which  began  in  Phase  1  (where 
time  =  today)  with  a  pre-determined  (i.e.,  “yesterday’s”)  FOPS  plan  implemented  in  the 
distributed  dynamic  decisionmaking  (DDD)  simulation.  The  DDD  is  a  simulation  tool  that  was 
used  to  display  locations  of  assets.  The  goal  during  Block  1  was  to  plan  for  Blocks  2  and  3.  The 
plan  for  Block  3  was  modified  in  Block  2  as  well.  FOPS  obtained  updated  information  from 
COPS  throughout  the  experiment  and  planned  for  the  next  two  Blocks.  The  FOPS  planning 
update  was  briefed  to  the  MOC  Director  and  submitted  as  a  new  plan  at  the  end  of  Block  1.  The 
plan  for  Block  2  was  implemented  at  the  tactical  level  in  DDD  at  the  beginning  of  Block  2. 

Block  2  was  the  second  2-hour  set  of  the  experimental  task.  Note:  Block  2  began  with  the 
implementation  of  the  FOPS  participants’  plan  produced  in  Block  1.  FOPS  planned  for  Blocks  3 
and  4.  The  plan  for  Block  4  was  modified  in  Block  3  as  well.  At  the  beginning  of  Block  2,  an 
emergent  threat  was  announced  such  that  the  MOC  would  be  assuming  responsibility  for  a 
second  area  of  tasks,  with  no  increase  in  resources  to  handle  the  load. 

Block  3  comprised  the  third  2-hour  set  of  the  experimental  task.  The  most  recent  Block  3  plans 
from  FOPS  were  implemented  to  begin  Block  3.  FOPS  planned  for  Blocks  4  and  5.  Block  5 
plans  were  not  implemented  in  this  experiment.  A  briefing  with  the  MOC  Director  was  held  at 
the  end  of  Block  3. 

Block  4  was  the  fourth  2-hour  set  of  the  experimental  task.  The  most  recent  Block  4  plans  from 
FOPS  were  implemented  to  begin  Block  4. 

Commander’s  Update  Briefings:  The  MOC  Director  (played  by  an  experimenter)  presented  the 
Commander’s  Update  Briefing  at  the  beginning  of  each  experimental  session  where  he  reviewed 
the  scenario,  including  the  Time-Phased  Force  Deployment  Data  (TPFDD)  -  the  arrival  schedule 
of  forces  into  theater,  and  the  task  graph.  Figure  1  presents  a  sample  task  graph  used  for  the 
Commander’s  update  brief  where  days  correspond  to  blocks.  Commander’s  guidance  for  task 
execution  was  given  to  indicate  task  priorities  and  key  prerequisites  for  day  T+l  (i.e.,  tomorrow) 
and  T+2  (i.e.,  day  after  tomorrow)  planning. 
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You  are  here  today  ^rea  g  Primary  Tasks 


Measures 

Measures  included  performance-focused  measures,  measures  of  perceived  workload,  information 
need  and  availability,  team  process,  and  communication  effectiveness.  We  report  on  several  of 
these  measures  here.  Additional  analysis  is  ongoing  and  will  be  included  in  the  presentation. 

Experimental  Design 

Decision-aid  tool  availability  was  manipulated  as  our  primary  independent  variable  as  the 
objective  for  this  research  is  to  gain  insight  into  the  advantages  and  limitations  associated  with 
integrated  versus  isolated  planners.  Decision-aid  tool  availability  had  two  levels:  (1)  whether  the 
FOPS  team  conducted  planning  using  the  full  FOPS  tool  -  where  players  could  see  planning 
progress  on  others’  tasks,  or  (2)  a  reduced  FOPS  tool  was  used  which  did  not  facilitate 
coordination  across  planners,  that  is,  the  planner  could  only  see  his/her  own  tasks.  In  the 
reduced-tool  condition,  when  planners  submitted  their  plan  to  the  subordinate  task  forces,  via  the 
agent  included  in  the  tool,  the  agent  informed  the  planner  of  the  asset-to-task  mapping  it  would 
recommend,  and  indicated  whether  its  earlier  recommendations  to  other  planners  included  those 
same  assets.  This  condition  subsequently  required  communication  between  planners  to  resolve 
asset-to-task  conflicts  before  submitting  a  joint  plan).  [Note,  the  FOPS  planning  tool  and  the 
intelligent  agent  are  described  in  the  next  section.] 

Materials  and  Scenario 

A  number  of  scenario  injects  were  crafted  in  the  form  of  casualty  reports  (CASREPS)  and 
situation  reports  (SITREPS)  to  cause  changes  regarding  the  planning  tasks  over  the  four  time 
blocks.  These  included  changes  in  asset  availability  that  occurred  over  the  course  of  the  scenario 
to  reflect  real-world  situations,  e.g.,  a  catapult  on  the  carrier  goes  down,  an  aircraft  system  incurs 
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a  problem,  or  a  missile  launcher  is  temporarily  reduced  to  50%  capability.  All  information 
arrived  on  an  a-periodic  basis,  at  various  time  intervals. 

Updates  were  sent  by  COPS  (played  by  a  confederate,  or  the  white  cell)  and  passed  to  FOPS 
regarding  the  following:  (1)  performance  on  individual  tasks  (accuracy  and  percentage 
completed)  for  that  day  and  (2)  reports  on  asset  capabilities,  such  as,  when  systems  were  back  up 
to  full  capability. 

Future  Operations  (FOPS)  Planning  Task 

FOPS  planners  were  charged  with  translating  the  mission  and  selecting  a  course  of  action  (CO A) 
to  instantiate  a  plan  for  that  mission.  They  considered  available  forces,  to  request  support  for 
execution  of  the  mission  tasks  for  which  they  were  responsible  (from  subordinate  task  forces 
either  as  the  primary  task  force  responsible  for  execution  of  the  task,  or  as  in  a  secondary  role  as 
a  supporting  task  force).  Then  they  determined  how  they  could  meet  their  planning  needs  within 
the  next  24  hours,  and  48  hours,  and  assigned  task  forces  as  either  primary  or  supporting  for  the 
given  tasks.  FOPS  also  monitored  the  information  received  in  the  CASREPS  and  SITREPS  to 
ascertain  the  implications  of  this  new  information. 

FOPS  Planning  Tool 

A  planning  tool  developed  to  facilitate  the  planning  process  was  provided  to  all  four  FOPS 
players.  Our  MOC-2010  experiment  utilized  analytical  models  in  a  relatively  new  and  expanded 
manner  by  embedding  analytical  models  within  an  on-line  decision  aid  for  asset  allocation  and 
planning.  This  decision-aid  module  was  developed  and  implemented  by  the  University  of 
Connecticut  (Mandal,  Han,  Pattipati,  &  Kleinman,  2010;  Han,  Mandal,  Bui,  Martinez,  Sidoti, 
Pattipati,  &  Kleinman,  2010).  This  work  presents  a  step  forward  in  the  design  of  decision  aids 
(and  agents)  for  human-in-the-loop  planning  activities. 

Figure  2  depicts  the  modeling  formalism  used  to  match  tasks  to  assets.  This  was  accomplished  by 
quantifying  the  amount  of  warfare  capabilities  required  to  accomplish  each  of  the  tasks  in  the 
scenario  and  quantifying  the  amount  of  warfighting  capability  provided  by  each  Task  Force.  For 
example,  in  Figure  2  we  see  conducting  Task  Tl,  air  early  warning  (AEW)  in  Area  A,  requires  5 
units  each  of  C2,  air  warfare,  and  ISR-air,  and  4  units  of  ISR-ground  capability.  Similarly,  the 
amount  of  each  of  the  warfighting  capabilities  provided  by  each  Task  Force  is  specified.  (These 
values  were  obtained  through  discussions  with  subject  matter  experts.)  FOPS  planners  assigned 
each  task  to  a  Task  Force,  and  also  specified  performance  goals  and  priorities  for  accomplishing 
each  task.  These  requirements  were  then  “submitted”  to  the  subordinate  Task  Forces  by  entering 
them  into  the  tool.  The  Task  Forces  then  determined,  via  the  agent,  how  best  to  utilize  their  assets 
to  meet  the  specified  performance  goals. 
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Task  Requirements  TF  Capabilities 


TASKS  AND  ASSETS  ARE 
REFERENCED  TO  THE 
SAME  SET  OF  SELECTED 
WARFARE  CATEGORIES 

•  Rik\  Requirement  k  of  task  /' 

•  rjk\  Capability  k  of  asset/TF  j 


Values  (resource  vectors)  are 
readily  obtained  via  discussions 
with  SMEs  or  Fleet  personnel 


TF  LEVEL  CAPABILITIES  ARE  ROLL-UPS  OF  THE  INDIVIDUAL  ASSET  CAPABILITIES 


•  FOPS  ASSIGNS  EACH  TASK  TO  A  TF,  with  PERFORMANCE  GOALS  and  PRIORITIES 

-  One  primary  TF,  with  others  assigned  as  supporting  in  one  or  more  warfare  areas 

-  The  TFs  determine  how  to  best  utilize  their  assets  to  meet  performance  goals 


Figure  2.  Modeling  Formalism:  Matching  Tasks  to  Assets. 

The  FOPS  planning  tool  is  an  interactive  computer-based  tool  that  provides  the  ability  for  the 
planner  to  assign  mission  tasks  to  subordinate  task  forces.  Players  were  given  a  set  of  constraints 
they  needed  to  consider  during  the  planning  process.  These  constraints  included  the  following: 

(1)  no  subordinate  task  force  (STF)  can  have  primary  responsibility  for  more  than  four 
concurrent  tasks,  (2)  a  STF  can  have  a  supporting  role  for  no  more  than  two  concurrent  tasks, 
and  (3)  there  can  be  no  more  than  two  supporting  TFs  on  any  given  task.  Once  a  FOPS  planner 
had  designated  STFs,  as  either  supported  (primary)  or  supporting  (secondary)  for  all  tasks  for 
which  he/she  was  assigned  responsibility,  the  plan  was  “submitted.”  An  agent-based  Task  Force 
algorithm  was  then  run  to  determine  assignments  based  on  the  expected  performance  levels 
designated  by  the  FOPS  players  for  all  tasks,  similar  to  the  way  planning  is  conducted  at  the 
operational  level  of  war. 

Additional  details  of  the  FOPS  planning  tool  are  described  in  Appendix  A.  Figure  3  depicts  the 
Summary  screen  that  is  one  of  several  displays  included  in  the  FOPS  planning  tool.  All  known 
tasks  are  shown  on  this  Summary  screen,  including  those  that  have  ended  or  are  not  yet  doable  or 
begun.  Those  tasks  that  are  not  relevant  for  that  day  are  shaded  out.  It  was  known  a  priori  when 
tasks  would  begin  and  (nominally)  when  they  would  end.  The  Summary  screen  provides 
functionality  for  the  following  processes:  (1)  submitting  a  preliminary  plan  to  the  subordinate 
Task  Forces  (TFs)  for  their  review;  (2)  finalizing  the  plan  that  will  be  briefed  to  the  MOC 
commander  at  the  end  of  the  day;  and  (3)  alerting  a  FOPS  player  when  new  (asset)  information 
has  arrived. 
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Finalizing  the  plan.  Finalize  means  to  send  out  the  actual  operational  orders  to  the  TFs.  Finalize 
is  only  valid  for  day  T+l  and  T+2  and  was  done  by  FOPS  only  at  the  end  of  the  lab  session.  The 
plan  for  today  (day  T)  is  already  being  executed,  and  so  no  changes  were  allowed  by  FOPS. 

Alerting  a  FOPS  planner  when  new  information  has  arrived.  When  any  asset  parameters 
have  changed,  COPS  pushed  information  to  FOPS  planners  via  CHAT.  The  FOPS  planner  used 
the  Asset  Status  screen  to  import  this  new  information  on  assets.  When  asset  capabilities  had 
changed,  the  results  that  have  been  provided  by  the  agent  models  may  no  longer  be  valid.  Thus, 
the  players  needed  to  resubmit  the  plan  to  the  TFs  for  adjustment,  taking  into  account  the 
changed  asset  capabilities. 

Assignment  Screen.  The  Assignment  screen,  depicted  in  Figure  4,  is  the  main  screen  that 
individual  planners  used  to  perform  their  job  of  assigning  tasks  to  primary  and  supporting  TFs  on 
specific  days  (T+l  and  T+2).  Tasks  were  assigned  one  at  a  time.  In  using  this  screen  the 
relevant  task  was  first  selected  via  a  drop-down  list,  where  only  the  relevant  tasks  for  the  day  in 
question  were  on  this  list.  (A  user  could  also  reach  this  screen  by  clicking  the  Task-ID  button  on 
the  Summary  screen  wherein  the  selected  task  would  be  brought  up  directly.)  Associated  with 
the  selected  task  is  shown  the  (fixed/  given)  task  priority,  and  the  decision  maker  (DM) 
responsible  for  planning  that  task  on  the  relevant  day.  The  desired  performance  goal/target  for 
the  day  is  input/changed  by  the  players  via  a  drop-down  list.  The  Expected  by  TFs  performance 
numbers  is  provided  via  the  TF  agent  models  only  after  an  overall  plan  has  been  submitted  to  the 
TFs  for  review  (via  the  Summary  screen). 

Assigning  a  task  involves:  (a)  selecting  a  primary  TF,  and  (b)  selecting  up  to  two  supporting  TFs 
-  each  one  possibly  supporting  in  up  to  two  warfare  areas.  Selecting  a  primary  TF  is  via  a  radio 
button,  so  that  there  can  only  be  a  single  primary  TF.  A  primary  TF  can  apply  any  or  all  of  its 
warfare  capabilities  to  the  task,  depending  on  the  allocation  decisions  reached  by  the  agent 
models.  Any  TF  not  designated  as  primary  can  be  a  supporting  TF.  The  selection  of  up  to  2 
supporting  warfare  areas  per  supporting  TF  is  via  a  drop-down  list  that  contains  all  of  the 
warfare  areas  as  well  as  “none”. 


13 


1  MOC-Experiment '  Team  A  '  Integrated  #  Day  0 

mmm 

File  Help  CurrentTimePeriod T=  0 

14 


The  Total  capabilities  lines  present  the  following,  by  warfare  area:  Possibly  (potentially)  applic¬ 
able  is  the  total  of  (1)  all  capabilities  in  the  Available  for  this  task  line  for  the  primary  TF,  plus 

(2)  all  capabilities  in  the  selected  warfare  areas  for  every  supporting  TF  as  given  in  their 
Available  for  this  task  line.  Clearly  this  is  an  optimistic  assessment  as  it  does  not  consider  other 
DMs’  actions. 

These  numbers  change  on-line  as  the  DM  makes  changes  to  the  task  assignment.  Last  Allocation 
by  TFs  is  the  amount  of  different  resource  capabilities  that  the  agent  models  last  assigned  to  this 
task,  across  all  TFs.  These  numbers  will  only  change  after  a  new  plan  is  sent  to  the  TFs  for 
review.  Shortage  is  the  difference  between  the  Last  Allocation  by  TFs  line  and  the  Estimated  task 
requirements  line. 

There  are  a  number  of  constraints  that  must  be  satisfied  in  the  overall  task-to-TF  assignments. 
These  constraints  require  DMs  to  collaborate  in  their  distributed  task  planning  and  assignment 
actions  include  the  following:  (1)  a  TF  cannot  be  primary  on  more  than  three  tasks  on  a  given 
day;  (2)  a  DM  cannot  select  any  supporting  TFs  on  a  task  until  a  primary  TF  has  been  selected; 

(3)  a  TF  cannot  be  supporting  on  more  than  three  tasks  on  a  given  day;  (4)  there  can  be  no  more 
than  two  supporting  TFs  on  any  given  task;  and  (5)  some  primary  assignments  of  a  TF  are  fixed 
and  cannot  be  changed:  a)  for  T+l  any  task  that  had  been  assigned  in  the  prior  day’s  T+2.  b)  for 
T+2  any  task  that  is  still  ongoing  from  the  previous  day  -  i.e.,  if  a  TF  was  primary  on  day  n  it 
must  continue  to  be  prime  on  day  n+1 .  Basically,  primary  assignments  once  made  on  a  task 
cannot  be  changed.  But  any  supporting  assignment  can  be  changed  as  needed. 

The  tool  checked  and  verified  (using  information  as  currently  posted  on  the  Summary  page)  that 
all  constraints  as  noted  above  were  satisfied  and  then  placed  the  new  task  assignment 
(performance  goals,  supported-supporting  relations)  into  the  table  on  the  Summary  page.  If  some 
constraints  were  not  satisfied,  a  pop-up  window  will  advise  the  DM  of  the  problem. 
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Summary _ Task  Status  Asset  Status  Assignment 

I  DAYT  |  DAY T  +  1  |  DAYT  +  2  | _ 


TASK: 

F  ^ 

TASK  PRIORITY: 

??  | 

RESPONSIBLE  DM. 

7? 

DESIRED  [??]  : 

EXPECTED  BY  STFS 

1  ^  1 

Qtaskpreq.status 

This  assignment  has  not  been  vetted  by  TF's 

C2 

STRK 

AW 

BMD 

CMD 

SUW 

USW 

MIW 

ISR_a 

ISR_s 

ISR_g 

BDA 

newl 

new2 

Primary  Supp^l 

5upp-2 

Est.Task  req's 

?? 

?? 

77 

77 

77 

7? 

7? 

77 

7? 

7? 

7? 

7? 

7? 

77 

TF-A 

|none 

|none 

Total  Capabilities 

14 

17 

33 

21 

22 

20 

16 

13 

24 

23 

2 

6 

0 

0 

Avail,  forthis  task 

14 

17 

33 

21 

22 

20 

16 

13 

24 

23 

2 

6 

0 

0 

Allocated  byTFs 

?? 

?? 

?? 

?? 

77 

7? 

7? 

77 

77 

7? 

77 

77 

77 

77 

TF-B 

|none  ^ | 

|none 

Total  Capabilities 

20 

6 

18 

0 

0 

1 

0 

0 

22 

9 

18 

15 

0 

0 

Avail,  forthis  task 

20 

6 

18 

0 

0 

1 

0 

0 

22 

9 

18 

15 

0 

0 

Allocated  byTFs 

?? 

?? 

?? 

7? 

77 

7? 

77 

77 

7? 

77 

77 

77 

77 

77 

TF-C 

0  [none  ■*  | 

|none 

Total  Capabilities 

6 

12 

0 

0 

0 

20 

40 

10 

4 

24 

4 

0 

0 

0 

Avail,  forthis  task 

6 

12 

0 

0 

0 

20 

40 

10 

4 

24 

4 

0 

0 

0 

Allocated  byTFs 

?? 

?? 

77 

77 

7? 

77 

7? 

7? 

7? 

7? 

77 

7? 

7? 

7? 

TF-D 

|none  » | 

|none 

Total  Capabilities 

13 

17 

29 

14 

18 

18 

17 

13 

20 

23 

2 

6 

0 

0 

Avail,  forthis  task 

13 

17 

29 

14 

18 

18 

17 

13 

20 

23 

2 

6 

0 

0 

Allocated  byTFs 

?? 

?? 

?? 

7? 

7? 

77 

77 

7? 

7? 

77 

7? 

77 

7? 

77 

TF-E 

|none 

| none 

Total  Capabilities 

17 

0 

0 

0 

0 

24 

8 

0 

11 

41 

17 

22 

0 

0 

Avail,  forthis  task 

17 

0 

0 

0 

0 

24 

8 

0 

11 

41 

17 

22 

0 

0 

Allocated  byTFs 

?? 

?? 

?? 

77 

77 

77 

77 

7? 

7? 

7? 

77 

77 

77 

7? 

TF-F 

1  nnnp 

1 nnnp  t 1 

Figure  4.  Assignment  Screen  in  the  FOPS  Planning  Tool. 
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RESULTS 


Each  of  six  teams  engaged  in  four  (4)  two-hour  sessions  (i.e.,  blocks)  of  the  experiment, 
discussed  here  in  relation  to  the  fictional  days  of  the  experimental  scenario  (i.e.,  Days  0,  1,2,  and 
3).  Days  0  and  1  consisted  of  training  the  participants  to  use  the  operational-level  planning 
software  with  guidance  from  experimenters.  Days  2  and  3,  considered  the  performance  periods, 
provided  greater  autonomy  to  participants  in  their  efforts  to  develop  plans  for  Future  Operations 
(FOPS).  The  experimental  goals  in  these  two  performance  periods  were  to  investigate 
differences  in  plan  quality  (performance),  workload,  and  communication  between  experimental 
conditions  that  differ  with  regard  to  shared  situation  awareness. 

Teams  in  the  integrated  condition  used  planning  software  that  enabled  full  situation  awareness 
regarding  the  assignment  of  task  forces  (TFs)  to  tasks  in  both  primary  and  supporting  roles. 

While  each  of  the  four  team  members  who  was  responsible  for  a  unique  set  of  tasks  in  a  specific 
planning  area  (Area  A  or  B)  during  a  specific  timeframe  [Time  T+l  (tomorrow),  or  T+2  (day 
after  tomorrow)],  all  members  in  this  integrated  condition  were  cognizant  of  the  current  planning 
allocations  by  everyone  within  the  team. 

In  contrast,  teams  in  the  isolated  condition  used  a  limited  form  of  this  planning  software,  in 
which  situation  awareness  was  limited  to  only  those  tasks  for  which  the  individual  team  member 
is  responsible.  Thus,  the  task  force  allocation  information  readily  available  to  integrated  team 
members  is  not  visibly  available  to  isolated  team  members,  and  must  be  obtained  through 
computer  chat-based  communication.  Results  from  between-condition  analyses  are  described 
throughout  the  remainder  of  this  section. 

Performance 

Performance  in  this  experiment  was  defined  as  the  quality  of  the  plan  produced  by  participants  in 
the  role  of  Future  Operations  (FOPS)  planners.  Four  specific  plans  were  submitted  at  the  end  of 
each  two-hour  performance  period:  plans  for  timeframes  T+l  and  T+2  in  both  Areas  A  and  B. 
Measures  of  planning  quality,  or  performance,  were  calculated  as  the  average  weighted  score  for 
each  task  (across  Areas  A  and  B,  within  a  specified  timeframe  -  either  T+l  or  T+2).  Task  scores 
contain  the  ratio  of  expected  plan  quality  levels  (either  accuracy  or  percent  complete)  to  those 
values  requested  by  Commander’s  Intent,  and  weighted  by  the  priority  of  the  task  in  relation  to 
other  tasks  (2  =  low  priority;  10  =  high  priority). 

It  was  expected  that  teams  in  the  integrated  condition  would  produce  higher  quality  plans  than 
the  isolated  condition  due  to  the  enhanced  shared  situation  awareness  provided  by  the  planning 
software.  Results  from  a  repeated  measures  analysis  of  variance  (ANOVA)  supported  this 
hypothesis  (see  Figure  5),  as  the  quality  of  plans  provided  by  the  integrated  teams  (M  = 

106.75,  SE  =  .73)  was  significantly  greater  than  plans  developed  by  teams  in  the  isolated 
condition  (M=  103.67,  SE  =  .73)  during  the  final  two  performance  periods  [F(  1 , 22)  =  8.91  ,p  < 
.01], 
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Integrated  Isolated 


Condition 


Figure  5.  Average  performance  between  conditions  across  the  two  final 

performance  periods. 

Additional  ANOVA  analyses  indicate  that  this  significant  pattern  of  differences  between 
conditions  exists  with  T+l  (tomorrow)  planners  [F(l,  10)  =  6.95 ,p  <  .03],  but  only  a  marginal 
difference  exists  among  T+2  (day  after  tomorrow)  planners  [F(l,  10)  =  3.40,/?  <  .10].  While  the 
integrated  teams  scored  consistently  higher  than  isolated  teams  for  both  T+l  and  T+2  planners 
(see  Figure  6),  there  was  a  greater  degree  of  variance  among  T+2  planners  (M=  107.17,  SE  = 
1.34)  than  among  T+l  planners  (M=  106.33,  SE  =  .72). 


■  Integrated 

■  Isolated 


Figure  6.  Average  performance  between  conditions  for  T+l  and  T+2  planners. 


Workload 

Workload  was  assessed  using  an  adapted  version  of  the  TLX  Workload  Scale  (Hart  &  Staveland, 
1988),  in  which  workload  is  measured  on  a  self-report  10-point  continuous  rating  scale  in  each  of 
five  distinct  dimensions:  mental,  time  pressure,  performance,  effort,  and  frustration.  This 
measure  was  issued  at  the  close  of  each  experimental  session  (four  times  per  team),  and 
demonstrated  sufficient  internal  consistency  ( a  =  .87). 

It  was  expected  that  workload  would  be  greatest  for  teams  in  the  isolated  condition,  as  the  lack  of 
shared  situation  awareness  provided  by  their  planning  tool  necessitates  an  increase  in  the  amount 
of  explicit  (communication-based)  status  updates  and  coordination  required  to  succeed  on  this 
interdependent  task.  Workload  was  also  expected  to  increase  as  the  experiment  progressed  from 
Days  0-3,  as  injects  that  limit  one’s  available  resources  increased  in  complexity  over  time. 
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As  expected,  results  of  a  condition  (2;  Integrated  vs.  Isolated)  by  session  (4;  Days  0-3)  between- 
within- subjects  ANOVA  (see  Figure  7)  indicate  that  the  average  overall  workload  reported  by 
isolated  team  members  (M=  42.89,  SE  =  3.75)  was  significantly  greater  than  workload  reported 
by  integrated  team  members  (M=  33.69,  SE  =  3.75)  across  all  four  experimental  sessions  (F(  \ , 
22)  =  3.01  ,p<  .05,  one-tailed). 
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Figure  7.  Average  workload  scores  between  conditions  across  all  experimental  sessions. 

Workload  was  also  compared  between  integrated  and  isolated  conditions  in  a  one-way  between- 
subjects  ANOVA,  showing  similar  significant  support  for  increased  workload  in  the  isolated 
condition  [F(l,  94)  =  5.61  ,p  <  .03].  Results  of  a  MANOVA  comparing  conditions  across  the  five 
dimensions  of  workload  showed  a  significant  difference  between  conditions  [Wilks'  Lambda  = 
0.167,  F(5,  90)  =  90.01,  p  <  .001].  Subsequent  univariate  analyses  showed  the  workload  ratings 
of  mental  pressure,  performance,  and  effort  to  each  be  significantly  greater  for  isolated  team 
members  (p  <  .01). 

Results  also  supported  the  hypothesized  increase  in  workload  as  the  experiment  progressed  (see 
Figure  8).  Specifically,  workload  ratings  provided  on  session  Day  0  (M=  29.08,  SE  =  2.48),  Day 
1  (M=  36.21,  SE  =  3.63),  Day  2  (M=  41.52,  SE  =  3.67),  and  Day  3  (M=  46.37,  SE  =  4.85)  were 
significantly  different  [F(3,  66)  =  5.86, p  <  .01],  and  increased  linearly  [F(l,  22)  =  8.50, p  <  .01]. 


Figure  8.  Linear  increase  in  workload  throughout  subsequent  days  of  the  experiment. 

In  addition  to  the  main  effect  for  session  shown  above,  isolated  teams  reported  greater  workload 
than  integrated  teams  in  each  block  (see  Figure  9).  Combined  with  the  findings  noted  above, 
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these  results  suggest  that  as  the  scenario  builds  and  the  interdependence  between  task  areas  A 
and  B  becomes  more  complex,  integrated  team  members  perceive  less  mental  workload  and 
demonstrate  more  effective  performance  than  isolated  team  members.  Thus,  providing 
additional  status-related  situation  awareness  can  increase  planning  performance  and  decrease 
workload  for  interdependent  planning  teams. 


Figure  9.  Greater  workload  for  isolated  vs.  integrated  team  members. 
Communication 

All  Chat  messages  sent  during  an  experimental  session  were  captured  electronically.  To  analyze 
the  chat  the  chat-file  for  each  experimental  session  was  printed  to  expedite  coding.  First  all 
acknowledgement  messages  (e.g.,  roger,  solid  copy,  confirm)  were  removed.  The  primary  metric 
was  the  percent  of  collaborating  messages  to  total  messages  sent.  Percent  collaborating  messages 
was  analyzed  using  a  Condition  X  Days  between-within-subjects  ANOVA.  As  we  can  see  from 
Figure  10,  the  percent  of  collaborative  messages  to  all  messages  increases  almost  linearly  from 
day  1  to  day  3  for  the  teams  in  the  isolated  condition,  whereas  the  percent  collaborative  messages 
for  the  teams  in  the  integrated  condition  falls  below  the  isolated  level  by  day  two  and  remains 
below  the  isolated  level  on  day  3.  This  interaction  was  significant,  F(2,  8)  =  4.36,  p  <=  .05.  It 
would  appear  that  the  need  to  discuss  the  collaborative  effort  increases  constantly  for  those  in  the 
isolated  condition.  But,  for  those  in  the  integrated  condition  once  they  establish  collaborative 
patterns  the  need  to  collaborate  falls  off  considerably. 


Figure  10.  Percentage  of  collaborative  chat  messages  by  condition  and  day. 

Additional  analyses  are  being  conducted  and  a  complete  discussion  will  be  included  during  the 
presentation. 
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APPENDIX  A:  Details  of  the  FOPS  Planning  Tool 


Summary  Screen 

The  Summary  screen  shows  the  task  assignment  actions  of  all  co-acting  and  interacting  players 
on  a  task-by-task  basis.  It  was  used  by  FOPS  to  iterate  a  plan  with  the  (subordinate)  Task  Forces 
for  either  day  T+l  (today  plus  1  day)  or  T+2,  and  then  to  finalize  the  plan.  [Note:  FOPS  used  the 
Assignment  screen  for  the  actual  planning  of  asset  allocation  for  all  tasks.] 

The  columns  on  the  Summary  screen  for  each  task  will  be  described  from  left  to  right.  ID  number 
(e.g.,  TA01)  -  this  button,  when  clicked,  will  take  the  decision  maker  to  the  Assignment  screen 
for  that  task  for  the  particular  day  in  question,  including  day  T.  Task  is  the  name  of  the  task  the 
information  on  this  row  describes.  Priority  is  the  priority  assigned  to  the  task,  ostensibly  by 
elements  external  to  FOPS  (that  is,  the  battlegroup  commander).  Decision  Maker  (DM)  is  the 
FOPS  DM  who  is  responsible  for  planning  the  task  (set  by  the  experimenters).  Note  that  any  one 
FOPS  DM  only  had  responsibility  for  a  subset  of  the  overall  tasks,  and  this  could  vary  from  day 
to  day.  Primary  TF  is  the  unique  primary  Task  Force  assigned  to  conduct  the  task.  Supporting 
TF  is  where  the  FOPS  planner  designates  up  to  two  supporting  Task  Forces  for  that  task  along 
with  specifying  those  warfare  areas  in  which  they  are  to  provide  support. 

Criteria  refers  to  the  desired  performance  criteria  associated  with  this  task,  which  is  specified 
either  in  terms  of  accuracy /  coverage  desired  or  percentage  of  task  complete  at  the  end  of  the 
day.  Typically,  a  task  that  persists  would  have  an  accuracy  or  percent  coverage  assigned  with  it, 
such  as  having  enough  aircraft  up  to  cover  75%  of  the  area  of  operations  for  a  surface 
surveillance  task.  The  task-specific  criterion  is  fixed  in  the  scenario,  i.e.,  whether  accuracy/ 
coverage  or  percent  complete  at  day’s  end  is  the  correct  criterion  for  a  given  task.  Desired 
Performance  specifies  performance  desired  for  this  task  by  FOPS  in  terms  of  either  accuracy  or 
percent  complete.  (These  numbers  are  entered  via  the  Assignment  page.)  Expected  Performance 
is  the  expected  accuracy  or  percent  complete  which  is  calculated  by  the  Task  Force  intelligent 
agent.  The  Task  Force  agent  models  provided  these  numbers  once  the  overall  plan  had  been 
submitted  to  the  TFs  for  review. 

Actual  performance  is  relevant  only  for  day  T  (i.e.,  today),  in  that  this  provides  feedback  to  all 
players  on  the  actual  performance  on  the  task.  This  information  came  via  COPS,  who  passed 
information  to  FOPS,  in  the  form  of  SITREPS  that  informed  FOPS  on  execution  of  the  mission. 
Confirm  Assignment  -  Since  different  FOPS  players  had  responsibility  for  planning  separate 
subsets  of  the  overall  tasks,  this  checkbox  was  used  to  synchronize  player’s  actions  before  the 
composite  plan  was  submitted  to  the  TFs.  If  any  change  was  made  to  the  task  assignment  (as 
saved  via  the  assignment  page)  this  box  became  unchecked.  Only  the  DM  responsible  for  the 
task  could  check  this  box.  Before  a  plan  could  be  submitted  to  the  TFs  or  finalized,  all  confirm 
boxes  must  be  checked. 

Submitting  the  plan  developed  by  FOPS  to  subordinate  Task  Forces  (TFs)  for  their  review  means 
to  “ask”  the  TFs  how  they  would  instantiate  the  composite  plan.  This  instantiation  was  done  by 
the  agent  models.  All  relevant  tasks  for  the  day  in  question  had  to  have  a  primary  TF  designated 
(in  the  Prime  STF  column),  with  a  priority  greater  than  zero,  and  a  desired  performance  number 
in  order  to  be  included  in  the  plan.  The  (new  or  revised)  performance  numbers  in  the  expected 
column  were  returned  from  the  agent  model.  These  performance  numbers  were  calculated  for 
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expected  accuracy  for  tasks  having  “infinite”  duration,  or  expected  percentage  completion  for 
tasks  having  finite  time  duration.  Submitting  the  plan  is  like  a  pushing  a  “what  if’  button  to  see 
what  kind  of  performance  can  be  achieved  given  all  the  competing  demands  on  limited 
resources. 

If  any  change  was  made  to  any  task  within  the  overall  plan  (including  priority,  desired 
performance,  expected  performance,  etc.),  the  plan  had  to  be  resubmitted  to  the  TFs.  Players  had 
indications  when  a  change  was  made  to  a  task  assignment  via:  a)  the  confirm  box  for  that  task 
became  unchecked  and  b)  the  expected  performance  for  that  task  showed  “??”.  If  Submit  had 
been  clicked,  further  clicking  of  this  button  (as  well  as  the  Save  button  on  the  Assignment 
screen)  for  the  particular  day  was  disabled  until  the  agent  algorithms  finished  and  returned  their 
results,  or  a  minimum  (fixed)  time  has  passed. 

Task  Force  Agent  Modal  Processing 

Agent  algorithms  were  designed  to  represent  the  TF  planning  cells  at  each  TF  level  for  each  TF. 
A  vector  contains  information  on  what  the  task  needs  based  on  the  latest  intelligence  and  when 
the  COPS  player  receives  new  intelligence  he/she  will  provide  updates  to  this  list  of  task 
requirements  based  on  these  inputs.  This  list  of  task  requirements  provides  the  best  information 
available  to  planners  on  what  is  needed  to  accomplish  the  tasks.  Based  on  the  assumption  that 
the  individual  TFs  are  coordinating  with  each  other,  the  agent  receives  the  plan  as  submitted  by 
the  FOPS  players  in  the  MOC  regarding  primary  and  secondary  assignments  of  TFs  to  specified 
mission  tasks.  The  agent  algorithm  determines  how  to  best  assign  individual  assets  (such  a  CG-1, 
UAV-4,  aircraft-6)  in  order  to  best  meet  desired  performance  levels  specified  by  the  MOC.  This 
is  accomplished  by  minimizing  differences  between  task  accuracy  (or  performance  completed) 
as  a  ratio  of  applied  resource  capability  to  the  estimated  task  resource  requirements.  The 
algorithm  minimizes  the  difference  between  what  is  thought  to  be  needed  to  accomplish  the  task 
and  desired  performance  as  a  weighted  sum  of  overall  tasks,  where  weight  equals  the  priority 
assigned  to  each  task. 

A  vector  is  produced  to  show  how  much  capability  will  be  provided  by  the  primary  TF  and  the 
supporting  TFs  to  each  task.  This  vector  is  presented  to  the  FOPS  players  to  show  which 
individual  assets  within  each  TF  are  assigned  to  which  task  and  how  much  of  that  asset’s 
capability  is  available  for  a  given  task.  This  includes  how  much  a  primary  TF  provides  to  each 
task  (a  primary  TF  can  support  up  to  four  tasks  at  any  time)  and  how  much  a  secondary  TF 
provides  to  each  task.  Agents  do  not  override  what  was  designated  by  FOPS  planners;  they 
merely  make  decisions  about  which  of  the  assets  to  apply  to  each  task.  For  example,  DDG-1  will 
use  two  units  of  C2  on  task  x,  and  two  units  of  C2  on  task  y.  (Note:  FOPS  does  not  receive  this 
level  of  detail  when  the  agent  presents  the  results  to  the  FOPS  planners.  But  this  information  is 
sent  to  the  DDD  so  it  can  show  which  assets  have  been  assigned  to  which  TF.) 

Assignment  Screen 

The  resource  vectors  shown  (across  task  and  TF  warfare  areas)  are  the  following:  The  Estimated 
task  requirements  is  the  current  “best  guess”  as  provided  via  intelligence  or  COPS  regarding  the 
capabilities  that  must  be  brought  to  bear  on  the  task  in  order  to  achieve  the  desired  performance 
goal.  These  numbers  were  calculated  by  the  software  from  stored  task  values  for  each  play 
session.  They  will  not  be  dependent  on  any  player  actions.  However,  they  will  change  on-line  if 
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the  FOPS  planner  makes  changes  to  the  desired  performance  goals,  e.g.,  if  a  player  increases  the 
desired  percent  complete  by  end  of  day  (or  increases  the  desired  accuracy)  for  that  task. 

For  each  specific  TF,  the  Total  capabilities  numbers  are  taken  directly  from  the  summation  line 
on  the  Asset  Status  page.  Thus,  it  behooves  the  players  to  continually  update  their  local  TF 
capability  data.  The  Available  for  this  task  line  is  the  previous  line  less  all  resource  allocations  - 
as  last  made  by  the  TF  agents  -  to  tasks  other  than  the  currently  selected  task.  If  no  composite 
plan  has  ever  been  sent  to  the  TFs  for  review  (via  the  Summary  page)  then  this  line  will  equal  the 
total  line.  An  “x”  in  a  resource  category  means  that  the  particular  TF  is  out  of  range  for  applying 
that  resource  (warfare  area)  to  the  selected  task. 

The  Allocated  by  TFs  line  shows  the  amount  of  resource  capability  that  the  TF  agents  last 
allocated  to  the  task.  Nominally  these  numbers  will  approximate  the  prior  line  in  those  warfare 
areas  relevant  (f  0)  to  the  selected  task.  If  no  composite  plan  has  yet  been  sent  to  the  TFs  for 
review  all  entries  will  be  “??”.  Also,  if  the  DM  made  any  change  in  the  assignment  of  the  task  to 
TF-X  this  line  will  either  show  all  “??”  or  be  shaded  out. 

The  Task  prerequisite  Status  button  is  only  needed  for  days  T+l  and  T+2  -  and  only  for  those 
tasks  that  are  scheduled  to  begin  on  the  day  in  question.  When  clicked  it  pops  up  a  window  that 
lists,  on  a  line-by-line  basis,  all  of  the  prerequisites  for  the  selected  task.  A  given  line  shows:  a) 
the  required  prerequisite  performance  criterion  (in  percentage),  and  b)  the  (actual  or  expected) 
performance  numbers  for  the  prior  day.  Thus,  for  tasks  starting  on  day  T+  1  we  show  the  actual 
performance  of  the  prerequisite  as  pulled  from  the  Summary  page  (if  no  entry  exists  yet  use  the 
estimated  performance  value  and  display  in  italic).  For  tasks  starting  on  day  T+2  we  show  the 
estimated  performance  from  the  day  T+l  Summary  page.  If  that  value  is  “??”  we  show  “TBD” 
in  the  pop  up  window.  If  a  task  has  no  prerequisites,  or  is  not  scheduled  to  begin  on  eitherT+1  or 
T+2,  the  button  is  shaded  out.  The  user  must  close  the  pop-up  window  before  any  other  actions 
can  be  taken. 
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